FreeBSD Impressions

I won't get into the systemd controversy but I'll give a breif synopsis of my experience with it. When I switched to Arch it was undergoing it's transition to systemd, so it was real bumpy ride for me for a while. I thought a lot of it was my lack of linux knowledge at the time, but in retrospect it was just a rough transition for the distro. A lot of things would randomly break for no reason. Every other system update would brick my system and I'd have to start from scratch again. Pulseaudio would just stop working for no obvious reason and I didn't even do anything. And I hated journalctl a lot! But I thought it was just one of those weird linux decisions that I couldn't fully appreciate because my lack of knowledge, so I just shrugged it off at the time.

And to this day I've had trouble getting the Environment directives working properly on service and nspawn units. I'm still not sure why it just doesn't work for no reason even though I've managed to get it working multiple times in the past seemingly doing the exact same thing. Either I forgot some detail or this too is some kind of weird bug.

Maybe it's a dumb idea to try to get the entire system layer depend on one giant incredibly complex program written in an unsafe language like C in an opaque way? I don't know. It kind of reminds me of the old ways of doing things where people would just brute force try a bunch of stuff until something worked and they would have no idea why, kind of like what it's still like to use wine, but would just post it somewhere like "this is how I got it working' — what? this is not good engineering. And I didn't even know that these frustrations were related to the systemd controvery until literally like a month or so ago, so I get the frustration.

My go-to distro is Arch Linux. I got into Arch because I wanted a linux that I could fully customize from the ground up. I like NixOS a lot with it's declarative approach but it's still too early for me to use it for everything — I use a lot of different tech and I had trouble getting things to compile and work, such as bucklescript. So I've decided to give Unix a try with FreeBSD.

These are my initial impressions of FreeBSD so far after working with it for the past month or so.

Documentation

FreeBSD's documentation has completely blown me away! I've really liked the Arch Linux docs for being really in depth, but FreeBSD takes it to another level. The handbook is very succinct and detailed, and the man pages are very in depth. I've never really had this feeling before: I get the sense that I can fully understand what my system is doing from the ground up. Arch kind of gets me sort of close to there but there's still a lot of magic. It's a great feeling.

Jails

Jails are really interesting. It's similar to a lot of the other container tech like nspawn, lxc and docker. The thing that really has stood out to me the most about it is it's granularity. Unlike the other container solutions out there, with jails I got a sense from the very start that I really knew what was happening with the container and what it was doing under the hood.

Comparison with other containers

For example, with docker, when you create a container there's a lot of stuff that's abstracted away and you don't necessarily know what it's doing. Like how does docker handle networking? surely it's creates some kind of bridge(s) because I see them with ifconfig. Which devices are available to it? how does it handle users under the hood? how is it bootstrapping these systems? the list goes on.

The same goes for nspawn. Which devices are available to the machine? how does it handle user details? I wrote an nspawn container a while back and I created a new user inside the container with the same username as my host running firefox, and somehow it automatically imported my host users profile! is it intended or is it a security hole? What can an nspawn container access? because this container was supposed to only have access to my Downloads folder. Why can I access some devices and not others? There's a lot of voodoo.

A comparably similar experience

Jails force you to have to make these decisions from the start, so you actually know what's going on. You might think that this extra upfront configuration is a downside, but no, not really — there are tools that give you those fancy abstractions you want like ezjail and iocage. Iocage feels a lot like docker or lxd to me, but I prefer manually setting up jails right now because I'm fascinated by the lack of magic.

Fine-grained security

Jails seem to me to give a lot more control over security.

Networking

When you create a jail, you have to explicitly declare which interface to connect to, which can be wired, or wireless, or a bridge, etc. You can have a jail with no access to any networks. Or you can easily set up a bridge that only applies to that jail with very restricted network access using a bridge and some simple pf rules (more on pf later), which is extremely useful if you want to containerize something like a browser, or IRC, or a mail client like Mutt.

Contrast this to something like docker/nspawn/lx* which is black box. Somehow the containers have access to the internet and probably all networking for no obvious reason. I'm sure you can dive into the internals or there might be some specific documentation on how to deconstruct this thing, but I seriously doubt it's as easy or straightforward as jails make it out to be, because the way jails and pf work is really as simple as I can imagine.

Devices

You can easily control which devices you want in your jail. For example, when you include mount.devfs as an option, it'll mount a devfs(5) using the devfs ruleset you provided in the devfs_ruleset parameter, or it'll use the default ruleset for jails which provides some basic device files like stdin, stdout and stderr.

Say for example you want to be able to enable sound in your jail. Well you can do that by allowing access to your sound device files for the sound (/dev/dsp*) and mixer (/dev/mixer*):

[devfsrules_desktop_jail=5] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add path zfs unhide add path 'mixer*' unhide add path 'dsp*' unhide

And remember the saying that in unix everything is just a file? well here you go! the mixers and sound device files are all files. This means that you can even have granular security on these devices. You could literally create a sound firewall if you wanted to in unix, that blows my mind!

A lot of things are still files in linux, but unfortunately it's starting to move away from this design with things like Pulseaudio which is more like magic again:

For OSS applications, PulseAudio provides the padsp utility, which replaces device files such as /dev/dsp, tricking the applications into believing that they have exclusive control over the sound card. In reality, their output is rerouted through PulseAudio.

Mounts

Unlike docker, jails actually give you the ability to give priviledged users inside a jail the ability to mount file systems, with a good amount of granularity. You can even provide an fstab file for more complex configurations.

Thin jails

A common technique is to create a thin jail, which is a jail with a very small footprint in terms of disk space. It's actually even smaller than the docker alpine linux images! The way this is done is to create a base jail skeleton, separating the read-only and read-write parts and mounting it as a nullfs file system using an fstab file. Ezjail and iocage has scripts to simplify this for you but you can do it yourself as a learning experience, or create your own simple script to do it.

Filesystem

One of the features I really like about jails is that the host has access to the jails filesystem directly. Nspawn is similar in this way. It's not a special volume like it is for docker. The decoupling of the state from the docker container is nice, but you can achieve the same thing easily using a nullfs mount.

Linux Compatibility Layer

There's a kernel module that you can enable to provide compatibility with linux binaries. There are actually two modules, one for both 32-bit and another for 64-bit. It actually works really well, I haven't had any issues running any linux apps.

Security

The BSDs have a long history of being used for network professionals, so there's a lot of security features built in. FreeBSD and OpenBSD are probably the best systems to use for networking and security, or at least I got that impression by reading the handbook and browsing the forums.

Pf

Pf is one of FreeBSD's firewalls, originally conceived by the OpenBSD team. It has a declarative configuarion file. It's actually a really simple and intuitive firewall, and a favorite among BSD users.

Here are some examples of common TCP and UDP services:

tcp_services = "{ ssh, domain, www, http, https }" udp_services = "{ domain }" block all pass out proto tcp to any port $tcp_services keep state pass proto udp to any port $udp_services keep state

The domain name is for DNS, the others are straightforward. All options are listed in /etc/services. The syntax reads like plain english. You can even set up the rules so that they listen to specific interfaces, which is really useful if you want to set up a specific ruleset, like for a bridge that a jail is using as an example.

The logs from pflog are binary, so you need to read them with tcpdump. To log firewall events you can listen to the interface that pflog creates, in my case pflog0. You can pipe it to a log file if you need to:

$ sudo tcpdump -n -e -ttt -i pflog0

You can decide which rules to log. In the ruleset above you can just append log, so the rule is like pass log ...

Try it out and visit a website. You can see that the dump produces logs that show the requests for the A and AAAA records that you're requesting from the DNS server!

Auditing

There's support for fine-grained security event auditing beyind just simple firewall logs. You can audit the following events and dump them in a log file:

• Supports user and kernel space events
• Login/logout events
• Command execution events
• File CRUD events
• Network events

Hardening

The FreeBSD installer gives you the option to provide some system and kernel hardening right from the get-go. You can install HardenedBSD which implements a lot of the same security features as SELinux (see the feature comparison). FreeBSD also implements file system access control lists and mandatory access control.

Kernel

FreeBSD does a lot of interesting things with the kernel that you don't get with Linux.

Customizable

It gives you some really nice tools to write your own custom kernel, debug it, and roll it back gracefully. This is really cool if you want to play around with kernel development or your own custom kernel hardening. On linux you're mostly stuck with QEMU for better or worse like I was when I was hacking on the android kernel.

Dtrace

There's actually a system-wide profiler for performance tuning that can also listen to kernel events called dtrace. I didn't even know this was possible! I have to try this out at some point.

Bhyve

There's a native (type 2) hypervisor for FreeBSD called bhyve. What makes this really interesting is that it's actually faster than KVM.

KVM uses QEMU in userspace which is a large general-purpose CPU emulator; this is a case where simpler is better - bhyve is streamlined and single-purpose

It supports VT-x but I haven't had a chance to try it. It might be fast enough to have it run Windows so that you can play games without dual booting or pulling your hair out with wine.

Filesystem

Consistency

So far I've noticed that FreeBSD is actually really consistent when it comes to the file system organization and hierarchy. Everything is where I would expect it to be. In contrast, often with linux I have to look up multiple places for stuff, especially config files which are all over the place.

And one of my favorite things: all logs are in /var/log/messages, no dreaded journald!

ZFS

FreeBSD uses ZFS instead of the Linux default of ext4. ZFS is a really cool. Basically it combines the concept of volume manager and file system into one — multiple file systems and drives can share the same pool of data (zpool). It also means that filesystems can be grown organically. If you add a new drive to your computer, it can be part of the same file system hierarchy instead having to mount it to a specific point with fstab. It also means that RAID is greatly simplied. And you get transparent distribution for free! a really interesting project demonstrating this potential is FreeNAS.

Networking

FreeBSD has excellent support for networking, as you've probably noticed from reading this blog post. Besides what I've already talked about, there are a few other interesting things I found out about:

HAST

This gives you transparent storage of the same data across several physically separated machines connected by a TCP/IP network. It's basically RAID1 but over a network! You can create a live backup of your entire file system.

CARP

This allows you to set things up so that multiple hosts share the same ip.

Drawbacks

With all of the niceties of FreeBSD, there are some drawbacks.

Hardware Support

One of the things I don't like is that the hardware support is kind of lacking in some areas. For example, currently FreeBSD doesn't have great support for getting the laptop to sleep when you close the lid — there's a bug with nvidia drivers that prevents it from working properly. I haven't tried nouveau though I should, maybe it'll work.

To be fair though, my laptop is like a decade old. And I have to manually install a broadcom-wl-dkms driver so that my wireless works properly on Arch Linux, which I don't have to do with FreeBSD.

Popularity

As mentioned earlier, FreeBSD is known for it's networking prowess. A lot of famous companies use FreeBSD:

Apache, Apple, Cisco, Citrix, Dell, Hacker, News, Juniper, McAfee, Netflix, Sony, The, Weather, Channel, Verisign, WhatsApp, Yahoo!, and Yandex. See the full list here.

Conclusion

I'm thoroughly impressed by FreeBSD. I'll write up more blog posts as I play around with it more in the future.